Editing interface

ABSTRACT

Associating media recording elements in a sequence with positions on a linear sensor, and presenting a corresponding one of the media recording elements to the user in response to the position of the users touch. In another general aspect, a media control apparatus which includes a linear actuator and a plurality of media storage areas ordered according to a time sequence. The linear actuator includes an actuation sensing area sensitive to user actuation at one of a plurality of positions along a length of the sensing area. The apparatus can be responsive to a position signal output of the linear actuator to access one of the media recording storage areas corresponding to a user-selected position. The apparatus can also include a linear display area mounted in parallel with the linear actuator.

This application is a continuation of application Ser. No. 08/215,131,filed Mar. 18, 1994 now abandoned.

CROSS-REFERENCE TO RELATED APPLICATION

This application is related to our design application entitled "EditingControl Panel" filed on the same day as the present application, andherein incorporated by reference.

1. Field of the Invention

This invention relates to controllers for editing sequences of storedmedia samples, such as video and audio material.

2. Background of the Invention

Post-production video, film, and sound editors must frequently look for,and play back, portions of a sequence of recorded material, such as adigitized video or audio recording. They can do this by using well knownbutton-type interfaces, which include a series of control buttons suchas: "play", "stop", "fast-forward", and "rewind". These allow the editorto sequentially move through the recorded material and play it back. Jogknobs, shuttle knobs, levers, and sliders are also available for movingthrough the material sequentially in order to find and play a givensequence.

In PCT Publication No. WO 93/21595, published Oct. 28, 1993, entitled"MEDIA COMPOSER INCLUDING POINTER-BASED DISPLAY OF SEQUENTIALLY STOREDSAMPLES," Eric C. Peters and Joseph H. Rice present a mouse-based methodof accessing video material. In this interface, a user can use a mouseto move a cursor to a location within a time line displayed on a videoscreen. This time line represents a sequence of recorded material, suchas one or more video sequences. By selecting a position in the timeline, the editor can display the image corresponding to that position.This interface is currently used in the "Media Composer" editing system,which is available from Avid Technology, Inc., of Tewksbury, Mass.

The time line in this system can display a cursor in the time line toindicate the currently selected position. It can also associate a seriesof representative frames of the video material with the time line, toassist the editor in finding a desired location within the material. Theuse of this time line in transition editing operations is discussed inU.S patent application Ser. No. 08/049,028, filed on Apr. 16, 1993,entitled "METHOD AND USER INTERFACE FOR CREATING, SPECIFYING ANDADJUSTING MOTION PICTURE TRANSITIONS", filed by Eric C. Peters, andJoseph H. Rice.

Digital audio editing operations have also been performed using amouse-based time line approach. These types of operations are describedin U.S. patent application Ser. No. 07/997,716, filed Dec. 31, 1992 andentitled "DISPLAY SYSTEM FACILITATING COMPUTER ASSISTED AUDIO EDITING",filed by Mark J. Norton. Further discussion of audio editing using timelines is presented in U.S. patent application Ser. No. 08/045,650, filedon Apr. 9, 1993, entitled "DIGITAL AUDIO WORKSTATION PROVIDING DIGITALSTORAGE AND DISPLAY OF VIDEO INFORMATION", filed by Peter J. Fasciano,et al.

A specialized mechanical user interface for video editing is presentedin U.S. patent application Ser. No. 07/932,703, entitled "MECHANICALUSER INTERFACE", filed by Henry Sharpe III et al. This interfaceincludes a slider which communicates with an Apple MacIntosh via theApple Desktop Bus standard. The user can actuate the slider to play backvideo material at an adjustable rate, or frame-by-frame. The aboveapplications are herein incorporated by reference.

SUMMARY OF THE INVENTION

Generally, the invention features associating media recording elementsin a sequence with positions on a linear sensor, and presenting acorresponding one of the media recording elements to the user inresponse to the position of the users touch. In another general aspect,the invention features a media control apparatus which includes a linearactuator and a plurality of media storage areas ordered according to atime sequence. The linear actuator includes an actuation sensing areasensitive to user actuation at one of a plurality of positions along alength of the sensing area. The apparatus can be responsive to aposition signal output of the linear actuator to access one of the mediarecording storage areas corresponding to a user-selected position. Theapparatus can also include a linear display area mounted in parallelwith the linear actuator.

This invention is advantageous in that it is easy to understand, evenfor editors who have never used a computer input device, such as amouse. With this system, users can quickly access video and audioinformation with simple, intuitive hand movements. Users do not need tohunt for a mouse on a desk, nor do they need to look for a cursor on acomputer screen and then move it to a control area. In some instances,the user may not even need to look at any control information on acomputer screen, allowing him or her to concentrate on the actualmaterial to be manipulated.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a perspective drawing of an editing control panel according tothe invention;

FIG. 2 is a block diagram of circuitry for use in connection with theediting control panel of FIG. 1; and

FIG. 3 is a state diagram illustrating operations using the editingcontrol panel of FIG. 1.

DETAILED DESCRIPTION

Referring to FIG. 1, an editing control panel 10 includes a jog-shuttlewheel 12, a fader bar 14, an X-Y image size/perspective joy stick 16,and an X-Y position/skew joy stick 18. The panel also includes first,second, third, fourth, and fifth banks of assignable buttons 20, 22, 24,26, 28, which may be defined by the user, or by system software. Alinear touch bar 30 sits across the panel, parallel to and just below athree-line linear screen 32 with a built-in driver. The fifth bank ofswitches is aligned parallel to, and above, the LCD screen.

Referring also to FIG. 2, the touch strip 30 can be a capacitive orresistive touch sensor with a built-in digital interface. This type oftouch strip is commercially available and operates by sensing acapacitance or resistance change on the strip due to contact by theuser's skin, Integral interface circuitry translates this capacitancechange into a digital value, which indicates where the user touched thetouch strip. The digital value generated by the touch strip may beprovided on a bus 34 to a local processor 36 located within the controlpanel 10.

The microprocessor 36 is also responsive to the switches in the fifthbank 28 via another bus 38, and it provides display information to thedisplay 32 via output lines 40. Similar connections supply signals fromthe jog-shuttle wheel 12, the fader bar 14, the two joy sticks 16, 18,and the first, second, third, and fourth banks of assignable buttons 20,22, 24, 26. As is well known, the processor is associated with storage35, such as semiconductor memory storage.

The processor has bidirectional input-output (IO) lines 31, which leadto a connector 33 on the rear of the housing of the control panel 10.These lines and connector are configurable according to any of a numberof serial or parallel interface standards, such as the well-known AppleDesktop Bus (ADB) standard, which is a bidirectional serial bus used toconnect keyboards, mice, and other peripheral devices to Apple personalcomputers. This connector can therefore be connected to an AppleMacIntosh personal computer 60 via a cable 37. The personal computer 60includes a processor 39, which is associated with storage 42, such assemiconductor storage 44, and disk storage 46. A display 48, andspeakers 50, 52 are also responsive to the processor.

The storage 42 of the computer holds various stored digital sequences43, 45. These can include code sequences, such as the computer'soperating system, applications, and drivers. They can also includerecording data, such as a series of digitized images. These can bestored in a series of storage areas 49 indexed by indexing informationstored in further storage areas 51.

Preferably, one of the applications run by the computer is the "MediaComposer" application 53 referenced above. Also stored in the computershould be a control panel device driver 47, which interfaces the signalsreceived from the control panel via the serial interface to the "MediaComposer" application as described below.

In operation, referring to FIGS. 1-3, the control panel 10 enters anawaiting input state 100. When the user touches a point on the touchstrip 30, it provides the microprocessor 36 with an indication that thiscontact has occurred, along with a value indicating where it occurred,via the bus 34 (step 102). The microprocessor receives this command,translates it to an event code, and forwards it to the personal computervia the serial interface. The microprocessor time-multiplexes this eventcode with other signals from the other controls, such as the switchesand joysticks.

The personal computer's processor 39 receives the touch command eventcode, and uses the driver 47 to translate it into a signal, which thecomputer's operating system interprets as mouse input event.Specifically, in the case of a user touching a point on the touch strip,the command is translated into an indication that the user had pressedthe mouse button while the cursor was at a corresponding point on thetime line displayed in the application. For example, touching the touchstrip 30 at its central point will be translated into an indication thatthe user had "single-clicked" with the mouse at the central point of thetime line.

In response to this translated event, the application causes theprocessor to use the indexing information 51 to retrieve a framecorresponding to the touched position from the stored video material,and display it on the display 48. In retrieving this frame, theprocessor accesses information indicating what the desired scale for thetime line and touch strip is. For example, if a ten minute digitizedvideo sequence is stored in the disk storage 46, and the touch barreports a value indicating that the user has touched it at it's center,the processor will retrieve and display the central frame in thatsequence, i.e., the frame at the five minute mark.

If the user lifts his or her finger from the touch strip 30 (step 106),the touch strip reports this to the microprocessor 36, which translatesand relays it to the computer's processor 39. The driver 47 presentsthis command to the processor as a release of a mouse button, and theapplication leaves the frame displayed on the display 48.

If instead of lifting his or her finger, the user slides it along thetouch strip 30 in a forward or reverse direction, the touch strip willprovide continuous updated position values to the microprocessor 36(step 108). The microprocessor and driver translate these values intoupdated mouse positions, with the mouse button being pressed. Theprocessor will respond to this sliding motion by successively displayingthe frames before or after the initially displayed frame. These aredisplayed at the same rate at which the user drags his or her finger,and this is known as a "scrub" mode. The driver or microprocessorprocedures may have to perform non-linear translations in order toachieve a pleasing playback response. Sampled audio information may alsobe retrieved from the storage by the processor and played through one ormore speakers 50, 52. This audio information may be played in synchronywith the displayed video information, or it may be played alone.

When the user then lifts his or her finger from the touch strip 30, itreports this to the microprocessor 36 (step 112). The control panel thenreturns to the awaiting input state 100, and the application leaves thecurrently displayed frame on the display, as described above. If,instead of lifting his or her finger (step 112), he or she stopsdragging it (step 111), the scrub mode will also stop with the currentframe displayed (step 104). Lifting his or her finger (step 106), willthen again return the control panel to the awaiting input state 100.

A cursor 54 is displayed in the screen 32, under control of themicroprocessor 36. The position of this cursor is in alignment with thelast finger position reported to the microprocessor by the touch strip.It remains aligned with this last position even after the user's fingerhas been lifted from the touch strip 30.

Different scales can be used for the touch strip 30. The user can makeadjustments to this scale by providing a scale command (step 114) to theapplication running on the computer processor 39. This scale command canbe provided to the processor through the computer's keyboard, its mouse,or through a button on the control panel assigned to this function.Actuation of the assigned button is detected by the microprocessor 36,which translates it into an event code that is sent to the computer 60.

In the computer 60, the driver 47 translates the received scale codeinto an application input event. The application responds to this eventby redefining the correspondence between the touch strip and thesequence of stored material. This can be done by updating the indexingmaterial to assign new points in the stored material to correspond tothe ends of the time line. In this way, the user may "zoom" in or out inthe material, or he or she may switch from working on one set of imagesto the next.

It is noted that the screen 32 has three lines, with the two bottomlines devoted to the cursor 54. The top line of the LCD screen can beused for displaying button label codes 27, 29. The computer 60 downloadsthese to the microprocessor 36, which translates them and provides themto the screen. A larger LCD screen could also be used, andrepresentative frames of the stored material could be displayed alopgthe touch strip in a similar way.

The above embodiment presents a control panel peripheral for use on theApple Desktop bus with a driver routine stored in an Apple MacIntoshcomputer. The system could also be implemented with other types ofcomputers, and other types of interfaces, such as IBM-PC parallel,IBM-PC serial, and SCSI interfaces. Analog recording or computergraphics recordings could also be displayed. Driver and microprocessorfunctionality could be implemented in the form of dedicated logiccircuitry, and other hardware/software tradeoffs are also possible.Instead of being touch sensitive, the touch bar could also be a pressuresensitive bar, or another type of linear, absolute and random accesscontrol. With a pressure sensitive bar, an additional dimension ofcontrol is available. For example, the pressure with which the userpresses upon the strip could be mapped to display brightness, audiovolume, or the like. Furthermore, although it is preferable to use astraight, linear touch strip, such a linear touch strip could includesome amount of curvature.

The present invention has now been described in connection with a numberof specific embodiments thereof. However, numerous modifications whichare contemplated as falling within the scope of the present inventionshould now be apparent to those skilled in the art. Therefore, it isintended that the scope of the present invention be limited only by thescope of the claims appended hereto.

What is claimed is:
 1. A media control apparatus, comprising:a pluralityof media storage areas ordered according to a time sequence, a linearactuator including a linear actuation sensing area sensitive to useractuation at one of a plurality of positions along the length of theactuation sensing area, and including a position signal outputresponsive to the actuation sensing area, means responsive to theposition signal output of the linear actuator, for accessing one of themedia recording storage areas corresponding to a user-selected one ofthe plurality of positions, and means responsive to a user scale commandto redefine the correspondence between the linear actuator and thesequence of stored material and thereby permit the user to zoom in orzoom out of the time sequence.
 2. The apparatus of claim 1 furtherincluding a linear display area mounted in parallel with the lineartouch sensitive strip.
 3. The apparatus of claim 2 further includingmeans responsive to the position signal output of the linear actuatorand for displaying a cursor in the screen corresponding to the one ofthe plurality of positions.
 4. The apparatus of claim 3 furtherincluding a series of buttons aligned in parallel with the screen, andwherein the means for displaying a cursor further includes means fordisplaying button labels proximate the buttons on the screen.
 5. Theapparatus of claim 4 wherein the means for displaying a cursor comprisea microprocessor; further including a serial interface responsive to thelinear actuation sensing area, and wherein the means for accessing isresponsive to the linear actuation sensing area through the serialinterface; further including a video display responsive to the mediastorage areas and to the means for accessing, and operative to displaycontents of the one of the media storage areas as a video frame; furtherincluding an audio output device responsive to the media storage areasand to the means for accessing, and operative to play contents of theone of the media storage areas as a sound; and wherein the means foraccessing include a driver responsive to the position signal output ofthe linear actuator and for translating a position signal from theposition signal output.
 6. The apparatus of claim 2 further includingmeans responsive to the position signal output of the linear actuatorand for displaying a cursor in the screen corresponding to the one ofthe plurality of positions.
 7. The apparatus of claim 3 wherein themeans for displaying a cursor comprise microprocessor.
 8. The apparatusof claim 3 further including a series of buttons aligned in parallelwith the screen, and wherein the means for displaying a cursor furtherincludes means for displaying button labels proximate the buttons on thescreen.
 9. The apparatus of claim 1 further including a serial interfaceresponsive to the linear actuation sensing area, and wherein the meansfor accessing is responsive to the linear actuation sensing area throughthe serial interface.
 10. The apparatus of claim 1 wherein the linearactuator is a capacitive touch bar.
 11. The apparatus of claim 1 furtherincluding a video display responsive to the media storage areas and tothe means for accessing, and operative to display contents of the one ofthe media storage areas as a video frame.
 12. The apparatus of claim 11further including an audio output device responsive to the media storageareas and to the means for accessing, and operative to play contents ofthe one of the media storage areas as a sound.
 13. The apparatus ofclaim 11 further including an audio output device responsive to themedia storage areas and to the means for accessing, and operative toplay contents of the one of the media storage areas as a sound.
 14. Theapparatus of claim 1 further including an audio output device responsiveto the media storage areas and to the means for accessing, and operativeto play contents of the one of the media storage areas as a sound. 15.The apparatus of claim 1 wherein the means for accessing include adriver responsive to the position signal output of the linear actuatorand for translating a position signal from the position signal output.16. The apparatus of claim 1 wherein the means responsive to a userscale command to redefine the correspondence includes means forassigning new points in the stored material to correspond to ends of atime line.
 17. The apparatus of claim 1 further including a lineardisplay area mounted in parallel with the linear touch sensitive strip.18. The apparatus of claim 1 further including a video displayresponsive to the media storage areas and to the means for accessing,and operative to display contents of the one of the media storage areasas a video frame.
 19. The apparatus of claim 1 further including anaudio output device responsive to the media storage areas and to themeans for accessing, and operative to play contents of the one of themedia storage areas as a sound.
 20. The apparatus of claim 1 furtherincluding means for changing the correspondence between the linearactuation sensing area and the plurality of media storage areas.
 21. Amethod of editing stored recordings, comprising:detecting a position ofa user's touch at one of a plurality of positions on a linear sensor,associating a plurality of media recording elements in a sequence withthe plurality of positions on the linear sensor, presenting acorresponding one of the media recording elements to the user inresponse to the position of the user's touch detected in the step ofdetecting, receiving a user command, redefining the correspondencebetween the plurality of media recording elements in the sequence withthe plurality of positions on the linear sensor based on the usercommand received in the step of receiving and thereby permit the user tothe a zoom in or zoom out of the material, again detecting a position ofa user's touch at one of a plurality of positions on a linear sensor,and presenting a second corresponding one of the media recordingelements to the user in response to the position of the user's touchdetected in the step of again detecting.
 22. The method of claim 21further including the step of displaying a cursor in parallel with thelinear touch sensitive strip.
 23. The method of claim 21 wherein thestep of detecting responds to a capacitance of the linear sensor. 24.The method of claim 21 wherein the step of detecting a position of auser's touch continuously detects the position of the user's touch, andwherein the step of presenting displays a series of the media recordingelements in a continuous fashion.
 25. The method of claim 24 wherein thestep of presenting displays the series of the media recording elementsas a series of frames in a video sequence.
 26. The method of claim 24wherein the step of presenting proceeds at a rate matching a rate ofmovement of the user's touch.
 27. The method of claim 21 wherein thestep of presenting displays the one of the media recording elements as avideo frame.
 28. The method of claim 21 further including the step oftranslating a result of the step of detecting into an event associatedwith a sensor that does not include a linear touch sensor.
 29. Themethod of claim 21 further including the step of displaying a cursor inparallel with the linear touch sensitive strip; wherein the step ofdetecting a position of a user's touch continuously detects the positionof the user's touch, and wherein the step of presenting displays aseries of the media recording elements in a continuous fashion; whereinthe step of presenting displays the series of the media recordingelements as a series of frames in a video sequence; and furtherincluding the step of translating a result of the step of detecting intoan event associated with a sensor that does not include a linear touchsensor.
 30. The method of claim 21 wherein the step of redefiningincludes the step of assigning new points in the stored material tocorrespond to ends of a time line.
 31. A method of editing storedrecordings, comprising:associating a plurality of media recordingelements in a sequence with a plurality of positions on a linear sensor,detecting a position of a user's touch at a first of the plurality ofpositions on the linear sensor, retrieving a first corresponding one ofthe media recording elements to the user in response to the position ofthe user's touch detected in the step of detecting a user's touch at afirst of the plurality of positions, wherein the step of retrieving afirst corresponding one of the media recording elements retrieves thefirst corresponding one of the media recording elements based on anassociation formed in the step of associating, detecting the release ofa user's touch on the linear sensor after the step of detecting aposition of a user's touch at the first of the plurality of positions onthe liner sensor, detecting a position of a user's touch at a second ofthe plurality of positions on the linear sensor after the step ofdetecting the release of the user's touch on the linear sensor, andretrieving a second corresponding one of the media recording elements tothe user in response to the position of the user's touch detected in thestep of detecting a user's touch at a second of the plurality ofpositions, wherein the step of retrieving a second corresponding one ofthe media recording elements retrieves the second corresponding one ofthe media recording elements based on the association formed in the stepof associating.
 32. A media control apparatus, comprising:a plurality ofmedia storage areas ordered according to a time sequence, a linearactuator including a linear actuation sensing area sensitive to a user'stouch at each of a plurality of positions along the length of theactuation sensing area, and including a position signal outputresponsive to the actuation sensing area, means for associating each ofthe plurality of positions with one of the plurality of media storageareas, and means responsive to the position signal output of the linearactuator, for accessing, upon detection of the user's touch on thelinear sensor at a one of the plurality of positions, the mediarecording storage area that corresponds to the one of the plurality ofpositions based on an association made by the means for associating, andretrieving contents of the media storage area that corresponds to theone of the plurality of positions, wherein the means is also for ceasingto retrieving contents of further media storage elements upon detectionof a release of the user's touch on the linear sensor and for againaccessing the media recording storage areas and retrieving contents offurther media storage elements based on the same association.
 33. Amedia control apparatus, comprising:an actuation sensing area sensitiveto user actuation at one of a plurality of positions along a length ofthe actuation sensing area, and including a position signal output,logic responsive to the position signal output of the actuator totranslate user actuation at user-selected ones of the positions intotranslated events for use by an application to access ones of aplurality of media recording elements ordered according to a timesequence such that the accessed media recording elements correspond tothe user-selected ones of the positions, and logic responsive to a userscale command to translate the user scale command into a translatedevent for use by the application to redefine the correspondence betweenthe actuator and the sequence of stored material and thereby permit theuser to zoom in or zoom out of the material.
 34. The apparatus of claim33 wherein the means responsive to the user scale command is alsooperative to permit the user to switch from one set of images to thenext.
 35. The apparatus of claim 33 further including a housing withrespect to which the actuator is mounted and a further including aplurality of buttons that are also mounted with respect to the housing.36. The apparatus of claim 35 further including a linear display areamounted with respect to the housing.
 37. The apparatus of claim 33further including a linear display area and further including meansresponsive to the position signal output of the actuator and fordisplaying a cursor in the linear display area corresponding to the oneof the plurality of positions.
 38. The apparatus of claim 33 wherein themedia control apparatus is operatively connected to a media editingsystem and further including a pointing device operatively connected tothe media editing system.
 39. The apparatus of claim 33 wherein theactuation area is part of a linear sensor.
 40. The apparatus of claim 33wherein the actuation area is part of a linear touch strip.
 41. Theapparatus of claim 33 further including a linear display area mounted inparallel with the linear touch sensitive strip.
 42. The apparatus ofclaim 41 further including means responsive to the position signaloutput of the actuator and for displaying a cursor in the linear displayarea corresponding to the one of the plurality of positions.
 43. Theapparatus of claim 42 further including a series of buttons aligned inparallel with the linear display area, and wherein the means fordisplaying a cursor further includes means for displaying button labelsproximate the buttons on the linear display area.
 44. The apparatus ofclaim 33 wherein the translated event codes translated from useractuation include an indication that contact has occurred and a valueindicating where it occurred.
 45. The apparatus of claim 33 wherein themeans responsive to the position signal and the means responsive to theuser scale include dedicated logic circuitry.
 46. The apparatus of claim33 wherein the means responsive to the position signal and the meansresponsive to the user scale logic include software.
 47. The apparatusof claim 33 further including means to provide an additional dimensionof control to the actuation sensing area.
 48. The apparatus of claim 33further including means for reporting a user actuation position on theactuation sensing area.
 49. The apparatus of claim 48 wherein the meansfor reporting a user actuation position on the actuation sensing areainclude a linear display area mounted with respect to the housing. 50.The apparatus of claim 48 wherein the means for reporting a useractuation position on the actuation sensing area include a time line.51. The apparatus of claim 48 further including random-access storagefor the media recording elements.
 52. The apparatus of claim 51 whereinthe storage includes a disk drive.
 53. A media control apparatus,comprising:an actuation sensing area sensitive to user actuation at oneof a plurality of positions along a length of the actuation sensingarea, and including a position signal output, means responsive to theposition signal output of the actuator to translate user actuation atuser-selected ones of the positions into translated events for use by anapplication to access ones of a plurality of media recording elementsordered according to a time sequence such that the accessed mediarecording elements correspond to the user-selected ones of thepositions, and means responsive to a user scale command to translate theuser scale command into a translated event for use by the application toredefine the correspondence between the actuator and the sequence ofstored material and thereby permit the user to zoom in or zoom out ofthe material.
 54. A media control method, comprising:detecting useractuation at one of a plurality of positions on a sensing area,translating the position detected in the step of detecting into atranslated event for use by an application to access ones of a pluralityof media recording elements ordered according to a time sequence,presenting a corresponding one of the media recording elements to theuser in response to the position of the user's touch detected in thestep of detecting based on an association between the plurality ofpositions and the plurality of media recording elements, receiving auser scale command, and translating the user scale command into atranslated event for use by an application to redefining the associationbetween the plurality of positions and the plurality of media recordingelements and thereby permit the user to the a zoom in or zoom out of thematerial.